🕷️️ Job Radar • SCRAPING

freelancer.com 2026-04-24 🟢

🔹 [Target] High-Volume Website Scraping Automation
👤 Client: 🇺🇸 Laredo, United States Member since 2015-11-07
💰 Price: $458 Average bid
🚩 Problem: Automate the collection of 90,000 symbols from public websites daily within a 4-6 hour window.
📦 Existing: Not specified

Specifications:

[Method] Implement an automated scraper using Python with Scrapy, Playwright, or Selenium for web navigation and data extraction.
[UI/UX] Not applicable (backend task)
[Stack] Python, Scrapy/Playwright/Selenium, BeautifulSoup, Pandas, Openpyxl
[Security] Handle CAPTCHAs, use rotating proxies, implement retries, and logging to resume from last point of failure.
[Format] Generate a single Excel workbook with rows for each symbol, embedding images or referencing file paths.

Workflow:

1. Define the scraping targets and URLs.
2. Develop the scraper using Scrapy/Playwright/Selenium for efficient page navigation and data extraction.
3. Implement image handling: embed images directly into Excel or save to a folder with references in the workbook.
4. Integrate CAPTCHA handling mechanisms, proxy rotation, and retry logic.
5. Schedule the scraper using cron jobs or task schedulers to run on weekdays at specific times.
6. Test the scraper for performance and reliability over multiple runs.
7. Deploy the solution in a production environment with monitoring.